Methods and systems for conceptually organizing and presenting information

ABSTRACT

Methods and systems for conceptually organizing and presenting information. A correlation of multiple users&#39; responses to the organization and presentation of information is used to determine an optimal organization and presentation of the information. For one embodiment of the invention, in the context of search engine result optimization, the user activity and/or user information of multiple users, during a search session, is correlated with queries to effect an evolving association between queries and the organization and presentation of documents. Systems in accordance with such embodiments employ the ability to store users&#39; activity over the entire search session, thus making possible the correlation of a number of different types of user activity and user information. The use of correlated user input allows such systems to provide relevant search results without the limitations imposed by the key-word-based systems of the prior art.

CLAIM OF PRIORITY

This application is related to, and hereby claims the benefit of, andincorporates by reference, provisional application No. 60/528,139, whichwas filed Dec. 8, 2003.

RELATED APPLICATIONS

This application is related to U.S. patent application Ser. No.“10/853,860”, entitled “METHODS AND SYSTEMS FOR PROVIDING A RESPONSE TOA QUERY”, filed on May 25, 2004, which is incorporated herein byreference. This application is related to U.S. patent application Ser.No. “10/917,721”, entitled “METHODS AND SYSTEMS FOR PROVIDING A RESPONSETO A QUERY”, filed on Aug. 12, 2004, which is incorporated herein byreference. This application is related to U.S. patent application Ser.No. “10/944,251”, entitled “METHODS AND SYSTEMS FOR PROVIDING A RESPONSETO A QUERY”, filed on Sep. 16, 2004, which is incorporated herein byreference.

FIELD

Embodiments of the invention relate, generally, to the field ofconceptually organizing information, and more specifically to theanalysis of the use of conceptually related information to effectorganization of information.

BACKGROUND

The capability of organizing information has grown along with theever-increasing availability of information. A vast source of availableinformation may be found on internet-related networks (e.g. the WorldWide Web (Web)) or other Internet sources. The Internet is an extensivenetwork of computer networks through which information is exchanged bymethods well known to those in the art (e.g. the use of TCP and IPprotocols, etc). The Internet permits users to send and receive databetween computers connected to this network. This data may include websites, home pages, databases, text collections, audio, video or anyother type of information made available over the Internet from acomputer server connected to the Internet. This information may bereferred to as articles or documents, and may include, a web page, dataon a web page, attachments to a web page, or other data contained in astorage device (e.g., database).

Making sense of such a very large collection of documents, and foragingfor information in such environments, is difficult without specializedaids. One such aid to assist in locating information is the use of keyterms. That is, the articles may include key terms representing selectedportions of the information contained in the article. These key termsare available over the Internet to other computers and permit theseother computers to locate the article.

To locate articles on the Internet, a user of a remote computer searchesfor the key terms using a search program known as a search engine.Search engines are programs that allow the remote user to type in one ormore search terms. The search engine then compares the search query withthe key terms from the articles and retrieves at least a portion of thearticles having key terms that match the search query. The search enginewill then display to the user the portion of the article such as thetitle. The user can then scroll through these retrieved portions of thearticles and select a desired article.

Early key-term search engines have exhibited serious drawbacks. Forexample, to increase exposure of a particular document, the documentprovider may use as many search terms as are possibly related to thearticle. In fact, some articles or search engines use every word in thearticle as key terms. As a result, search engines will retrieve manyarticles that are unrelated, or only peripherally related, to thesubject matter that the user desires to find through a combination ofsearch terms. Additionally, many users of such search engines are notskilled in formulating key-term search queries and produce extremelybroad searches that often retrieve thousands of articles. The user mustthen examine the excerpted information regarding each document to locatethe desired information.

This drawback was addressed by the evolution of search engines toinclude the organization of information based upon the search activityof one or more users. Such schemes rank results based upon a consensusof user preferences instead of document-oriented parameters (e.g.,text). One such scheme ranks documents according to an evolving scorebased upon the key terms used. That is, the documents receive arelevancy score relative to the key terms of the search query. As usersenter search queries and select documents from among the list ofdocuments the query produces, the relevancy scores of the documents areadjusted. The scores are used to organize the resulting list ofdocuments for subsequent searches. Such schemes typically baserelevancy, at least in part, on the number of “clicks” the documentreceived (i.e., the number of times a document was selected). Suchschemes, known generally as “popularity ranking schemes” or “clickpopularity schemes”, provide a search result list in which the highestranked documents are those that attracted and satisfied the greatestnumber of previous users. Moreover, click popularity schemes generateresults that reflect search context. For example, previous searchschemes would return documents containing all of the query terms, butwould not automatically exclude words that are not part of the query.Thus, a text-matching search for “Mexico” might return mostly resultsabout “New Mexico.” A click popularity scheme search will reduce sucherroneous results, as users seeking “Mexico” will generally refrain fromclicking pages about “New Mexico” and will tend to click those pagesthat they discern are most relevant to “Mexico,” thus raising therelevancy of the desired documents.

Basing relevancy on the number of clicks may lead to erroneous resultsover time as information related to the query terms changes. Forexample, for the particular query of “democratic frontrunner,” documentsreferring to the early-stage frontrunner Howard Dean may have beenselected numerous times in December 2003, but the user in March 2004entering that query may have been anticipating results for John Kerrywho was then leading. Additionally, top-ranked results generally receivedisproportionately greater use resulting in increasingly skewed searchresults in which the top-ranked results may never be displaced.

Some of these drawbacks have been addressed by search engines thatorganize information provided in response to queries using numerousfactors including time-based and use-based factors. For example, such ascheme may use the activity of previous users in response to particularqueries to adjust the relevancy of the query response documents. Suchuser activity may include the number of clicks in conjunction with thetiming of prior users' selections or use of particular information. Suchschemes may also consider where in a prior results listing a particulardocument was ranked when prior users selected it, actual versus expecteduse frequency of a document, and how the selected documents were used.

Yet, many drawbacks still exist in the current schemes. For example,current schemes do not address the problem of very rare queries forwhich sufficient user activity data has not been compiled. In suchcases, results may be poor or non-existent. Additionally, click resultsare dependent upon the quality and integrity of the data source. Currentschemes fail to account for the wide variations in data source quality.Moreover, current schemes are subject to spurious influences that mayaffect the integrity of the search results. One exemplary scheme, inaccordance with the prior art, attempts to address certain drawbacks byupdating search engine results based upon user activity. This scheme isdescribed in U.S. Pat. No. 6,421,675 entitled “Search Engine” which ishereby incorporated by reference herein, to provide a fuller descriptionof the prior art and clearly distinguish features of various embodimentsof the present invention.

SUMMARY

An embodiment of the invention provides a method in which a set ofconceptually related information is provided to a plurality of users.One or more responses regarding the presentation of the conceptuallyrelated information is received from each of two or more users. Theresponse from two or more users is then correlated and the presentationof the conceptually related information is modified based upon thecorrelated responses.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may be best understood by referring to the followingdescription and accompanying drawings that are used to illustrateembodiments of the invention. In the drawings:

FIG. 1 is a flow chart that illustrates a process in which theorganization and presentation of a set of conceptually relatedinformation is modified based upon a correlation of responses frommultiple users in accordance with one embodiment of the invention;

FIG. 2 is a flow chart that illustrates a process by which anassociation of stored content with a user query is effected based uponthe correlation of user activity and/or user information obtained duringa search session with similar information of other users in accordancewith one embodiment of the invention;

FIG. 3 is a block diagram that illustrates a system for organizing andpresenting conceptually related information and associating particularstored content with respective user queries in accordance with oneembodiment of the invention;

FIG. 4 illustrates the search log for each of three independent users inaccordance with one embodiment of the invention;

FIG. 5 is a flow chart that illustrates a process for providing morerelevant search results in accordance with one embodiment of theinvention;

FIG. 6 is a flow chart that illustrates a process for providing relevantquery suggestions in accordance with one embodiment of the invention;

FIG. 6A is a flow chart that illustrates a process for providing queryspelling correction suggestions in accordance with one embodiment of theinvention;

FIG. 7 is a flow chart that illustrates a process in which suggestedqueries are provided in accordance with one embodiment of the invention;

FIG. 8 is a flow chart that illustrates a process in which similarresults are provided in accordance with one embodiment of the invention;

FIG. 9 is a flow chart that illustrates a process in which informationis correlated to provide more relevant search results in accordance withone embodiment of the invention;

FIG. 10 is a flow chart that illustrates a process by which morerelevant search results are provided to a user based upon the user'slocation in relation to a location associated with a pick in accordancewith one embodiment of the invention;

FIG. 11 is a flow chart that illustrates a process by which morerelevant search results are provided to a user based upon the user'slocation in accordance with one embodiment of the invention; and

FIG. 12 is a block diagram that illustrates a digital processing systemin accordance with one embodiment of the invention.

DETAILED DESCRIPTION

Overview

Embodiments of the invention provide methods and systems forconceptually organizing and presenting information in which acorrelation of users' responses to the organization and presentation ofinformation is used to determine an optimal organization andpresentation of the information. For one embodiment of the invention, inthe context of search engine result optimization, the user activityand/or user information of multiple users, during a search session, iscorrelated with queries to effect an evolving association betweenqueries and the organization and presentation of documents. Systems, inaccordance with such embodiments, employ the ability to store users'activity over the entire search session, thus, making possible thecorrelation of a number of different types of user activity and userinformation. The use of correlated user input allows such systems toprovide relevant search results without the limitations imposed by thekey-word-based systems of the prior art.

In the following description, numerous specific details are set forth.However, it is understood that embodiments of the invention may bepracticed without these specific details. In other instances, well-knowncircuits, structures and techniques have not been shown in detail inorder not to obscure the understanding of this description.

Reference throughout the specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearance of the phrases “in one embodiment” or “in an embodiment” invarious places throughout the specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments.

Moreover, inventive aspects lie in less than all features of a singledisclosed embodiment. Thus, the claims following the DetailedDescription are hereby expressly incorporated into this DetailedDescription, with each claim standing on its own as a separateembodiment of this invention.

Process

FIG. 1 illustrates a process by which the organization and presentationof a set of conceptually related information is modified based upon acorrelation of responses from multiple users in accordance with oneembodiment of the invention.

Process 100, shown in FIG. 1, begins with operation 105 in which a setof conceptually related information is organized and presented tomultiple users by an information provider. For one embodiment, the setof conceptually related information is a gathering page containinginformation related to a general concept of interest to a particulargroup of users. Such a gathering page, may contain a number of relatedsub-concepts of any sort, including links to documents, directories,databases, spreadsheets, news items, audio, video, images, applications,advertisements, product descriptions, and reference information, as wellas links to lists, tables, trees, or catalogs of any of the above items,and links to other gathering pages, all of which may be collected fromany number of sources. For one such embodiment, multiple, distinctlyformulated, gathering pages are created that vary the organization andpresentation of the set of information, including the number, type,arrangement, and prominence of the related sub-concepts. In this way,the information provider attempts to anticipate an optimal way oforganizing and presenting the information.

At operation 110, the information provider receives responses from theusers in regard to the organization and presentation of the information.The users' responses may be in the form of selecting (or refraining fromselecting) portions of the information. For example, some users mayselect information of particular sub-concepts of a given gathering page,while not selecting others.

At operation 115, the received responses from multiple users arecorrelated. That is, a determination is made as to the extent to whichthe same statistically significant associations have been made byindependent users. For alternative embodiments of the invention, thecorrelation of responses may take many forms. Exemplary correlations forvarious user responses are described in more detail below.

At operation 120, the organization and presentation of the set ofconceptually related information is modified based upon the correlatedresponses. For example, a gathering page may be reformulated to moreadequately meet the needs of the an individual user, a group or class ofusers, or all users. In addition to modifying the presentation of theoriginally presented information, such reformulation may include addingor deleting information. For example, one or more sub-concepts may beadded to or deleted from a gathering page.

Though described generally above in relation to using user responses tooptimize the organization and presentation of information sets (e.g.,gathering pages), embodiments of the invention may be used to effect theassociation of user search engine queries to stored content (e.g., oneor more documents). That is, embodiments of the invention may be used todetermine search results in response to a particular query that are morerelevant (i.e., either generally more relevant or more relevant to aparticular user).

FIG. 2 illustrates a process by which an association of stored contentwith a user query is effected, based upon the correlation of user searchengine (USE) activity and/or user information obtained during a searchsession with similar information of other users in accordance with oneembodiment of the invention.

Process 200, shown in FIG. 2, begins at operation 205 in which USEactivity information and/or user information, during a search session,is recorded for several independent users. A search session includes anysequence of search engine actions (activities that can be recorded bythe search engine) of a given user. The USE activity may include theissuing of queries, the clicking of links on the search page leading tointernal or external data, the clicking of links on subsequent internalpages leading to internal or external data, and a return to the searchpage or any internal page subsequent to clicking an internal or externallink. The USE activity may be continuous or occur within a practicalduration period. That is, a time period may be specified that indicatestermination of a search session. For example, if an interruption inrecorded user activity exceeded a specified time, it may be practical toassume the search session was terminated. Subsequent user activity maybe viewed as a new search session.

At operation 210, the recorded USE activity information and/or userinformation for several independent users is correlated. Theinformation, which reflects entire search sessions for each user, may becorrelated in various ways in accordance with alternative embodiments ofthe invention. Exemplary correlations for various USE activity and/oruser information are described in more detail below.

At operation 215, the results of the correlation of the recorded USEactivity information and/or user information for several independentusers obtained from operation 210 are used to effect an associationbetween particular stored content and corresponding user queries. Suchassociation may provide a search result for a given query having greaterrelevance, in general, or to one or more particular users.

System

Embodiments of the invention may be implemented in a networkenvironment. FIG. 3 illustrates a system for organizing and presentingconceptually related information and associating particular storedcontent with respective user queries in accordance with one embodimentof the invention. System 300, shown in FIG. 3, illustrates a network ofdigital processing systems (DPS) 300 that include one or more serverDPSs, shown as server DPS 320 and a plurality of client DPSs, shown asclient DPSs 305–308. The DPSs of system 300 are coupled one to anotherand are configured to communicate a plurality of various types of storedcontent including documents, such as, web pages, content stored on webpages, including text, graphics, and audio and video content. Forexample, the stored content may be audio/video files, such as programswith moving images and sound. Information may be communicated betweenthe DPSs through any type of communications network through which aplurality of different devices may communicate, such as, for example,but not limited to, the Internet, a wide area network (WAN) not shown, alocal area network (LAN), an intranet, or the like. For example, asshown in FIG. 3, the DPSs are interconnected one to another throughInternet 310, which is a network of networks having a method ofcommunicating data, as described above, and is well known to thoseskilled in the art. The communication links coupling the server DPS toclient DPSs need not be a direct link, but may be indirect linksincluding, but not limited to, broadcasted wireless signals, networkcommunications, or the like. While exemplary DPSs are shown in FIG. 3,it is understood that many such DPSs are interconnected.

An embodiment of a digital processing system that may be used for theserver DPS 320 or the client DPSs 305–308, in accordance with anembodiment of the invention, is described below in reference to FIG. 12.

In accordance with one embodiment of the invention, a set ofconceptually related information, such as a gathering page, is presentedto several users via client DPSs 305–308. The conceptually relatedinformation may also be a search result in response to a user query,communicated from one or more of the client DPSs 305–308. Theinformation may take many forms, and may be, for example, a list ofweb-page URL addresses provided by a web-page developer. Once presentedon the client DPSs, the user makes some response to the presentation.For example, the user may conduct USE activity as described above. Theuser response and other user information, related to the presentation ofthe information to the user, is recorded and communicated to the userresponse/information correlation application 321. The application 321correlates the user responses and information of several users andmodifies the organization and presentation of the conceptually relatedinformation based upon the correlation results.

Data Analysis

Embodiments of the invention obtain USE activity information and/or userinformation and correlate this information to aid in defining relevancythrough the use of consensus choices of multiple users. This correlationanalysis includes the process of evaluating common actions, orinformation of multiple users, to identify statistically significantassociations. The terms “association” and “statistically significantassociation”, as used in regard to such embodiments, are defined asfollows. An “association” is any pairing of queries, terms, concepts,articles or other web data, or combinations thereof, which are madeexplicitly or implicitly, consciously or unconsciously, by a user duringa search session. An association may be indicated through the issuing ofqueries and/or the selecting of links (e.g., hypertext links) toqueries, terms, concepts, articles, or other web data. A statisticallysignificant association is an association that is probably notattributable to random occurrence. A correlation is recorded when astatistically significant association is made by two or more ostensiblyindependent users.

Embodiments of the invention provide search engine results with fargreater relevancy than prior art schemes by recording much moreinformation and more specific information (including USE activityinformation and/or user information), and effecting a more in-depthanalysis of the information.

One embodiment of the invention provides a system that creates andoperates on data files that maintain all USE activity information anduser information as shown in Table 1. (Tables used to describe variousembodiments of the invention are exemplary and do necessarily representthe actual data structures of embodiments of the invention.)

TABLE 1 Timestamp User ID Query Pick (URL) Jan. 1, 2003 00:00:00 U1 Q1P5 Jan. 1, 2003 00:01:00 U1 Q2 P1 Jan. 1, 2003 00:02:00 U1 Q2 P2 Jan. 1,2003 00:02:05 U1 Q2 P3 Jan. 2, 2003 00:00:00 U2 Q2 P4 Jan. 2, 200300:01:00 U2 Q2 P1 Jan. 2, 2003 00:02:00 U2 Q2 P4 Jan. 2, 2003 00:04:00U2 Q1 P2 Jan. 2, 2003 00:04:05 U2 Q1 P3 Jan. 3, 2003 00:00:00 U3 Q3 P3Jan. 3, 2003 00:04:00 U3 Q2 P1 Jan. 3, 2003 02:00:00 U3 Q3 P5 Jan. 4,2003 00:00:00 U4 Q1 * Jan. 4, 2003 00:06:00 U4 Q2 P4 (*= query with noassociated pick)

Table 1 shows a data file containing a number of data elements thatrecord the queries for a number of users at various times and the URL(pick) that was selected (clicked) subsequent to each respective queryfor each respective user. Such a data file, may contain numerous otherdata elements representing USE activity information and/or userinformation. Such data elements, may represent, for example, the displayrank of the result selected, the order the result was clicked by theuser during the session, the user IP address, geo-location of the IPaddress, etc.

This is in contrast to various prior art schemes in which, afterperiodic processing and loading of new data, simplified query-to resultpick (Q2RP) correlations were rolled up into a database table such asTable 1A. In accordance with one embodiment of the invention, suchinformation is not rolled up, but is maintained in log form, with dataelements indicating all of the USE activity information and userinformation.

TABLE 1A Query Pick (URL) Score Q1 P2 1 Q1 P3 1 Q1 P5 1 Q2 P1 3 Q2 P3 1Q2 P4 2 Q3 P3 1 Q3 P5 1

Table 1A is a simplified example of prior art data structure, which mayinclude score adjustment fields, as known in the prior art. For example,a score, which could be something as simple as the sum of the picks,could also be the result of a more sophisticated adjustment algorithm;the scores and adjustments could also be stored.

The prior art data structure of Table 1A was adequate for producingranked search results, but it represented the loss of a great deal ofraw information. As exemplified in Table 1A, prior art schemes did notrecord or analyze much of the USE activity information and userinformation available from a search session. This was due to a number ofreasons including limited storage, lack of any practical use for theinformation, and lack of appreciation of the application to which suchinformation could be applied.

In accordance with one embodiment of the invention, the need for suchscore adjustment fields is obviated by the extent of the recorded andstored information. That is, for such embodiments, no score adjustmentinformation is stored because any score needed can be calculatedparametrically based on the stored information. Moreover, parameters andalgorithms for calculating scores can be altered, as necessary, withoutaffecting stored data.

In accordance with various embodiments of the invention, the in-depthanalysis of the information depends upon the ability to recognize andrecord correlative data (including USE activity information and userinformation) between independent users. That is, by recording andanalyzing much more of the information, including up to all of theinformation of a user session, search results having a much higherdegree of relevancy can be obtained.

For example, consider a user session in which a user issues a series ofqueries and intervening picks. In general, the URLs picked prior to theissuing of query A will be irrelevant to query A, since users will oftenchange topics. Similarly, most picks occurring after a subsequent queryB, will be irrelevant to query A.

Further, consider a number of independent users who all record searchsessions containing query A. Such users can be expected to turn to avariety of unrelated topics before and after query A, and therefore, theirrelevant picks will be widely scattered over a huge number of URLs,each of which will garner very low scores reflecting their lack ofassociation with query A. Typically, only a very small number ofrelevant URLs that are picked by those users who do remain on thosetopics related to query A, will accumulate click popularity scores highenough to affect the re-ranking of the search results associated withquery A.

For example, imagine 1000 users who search for query A. Subsequently,900 of them then turn to unrelated queries B1 through B900. Theremaining 100 continue to search on the variations of the original topicand turn to related queries A1–A9. The results generated by the Bqueries will each collect a pick or two that will be associated withquery A, but the results generated by related queries A1–A9 willaccumulate ten times as many picks on average.

For one embodiment of the invention, a requirement that a URL, followinga subsequent query, must be picked at least twice to be correlated to anoriginal query may be imposed. Such a requirement, will eliminate manyof the picks of the B queries erroneously associated with query A.

In accordance with one embodiment of the invention, a correlationanalysis is provided that relies on statistically large samples torecognize multiple relevant associations. For such embodiments, priorart restrictions on the associations analyzed, and arbitrarycompartmentalization of the data, are reduced to increase relevancy.That is, recording and analyzing much more of the user sessioninformation also allows for many more relevant types of associations tobe analyzed.

Exemplary Use Activity Correlations

In accordance with one embodiment of the invention, a system is providedthat uses one or more of a small set of basic correlations, andcombinations thereof. In general, any number of correlations may bedetermined and used to effect increased relevancy of search results, orother objectives, in accordance with various embodiments of theinvention. Some exemplary correlations will be discussed, in detail,below.

Query-to-Pick

A query-to-pick (Q2P) correlation associates a query with a pick. Whenmultiple independent users make the same association, that is acorrelation candidate. When a search engine returns a result in responseto a query and a user picks that result, this is a special case of sucha correlation (Q2RP). In effect, the search engine algorithm replaces asecond independent user. In accordance with one embodiment of theinvention, the Q2P correlation associates a query with all picks in auser session. This is in contrast to prior art schemes that terminatedassociation of a given query with picks upon issuance of a subsequentquery.

With Q2P, all picks recorded during a user session are associated with agiven query issued during that user session. For one embodiment, a scoreis assigned to each association, based upon various factors, includingthe time between query and pick, the number of intervening queriesand/or picks, and the order of queries with respect to picks.

In addition, each association's score can be adjusted, based uponwell-known factors, including rank of the pick in the result list at thetime of association, duration of the pick (interval until next knownuser action), age or order of the association (relative to older ornewer associations), and age of the first known instance of association.

Each user session can be of infinite duration. In a practicalapplication, a reasonable time limit, or limit on intervening actions,should be imposed beyond which no relationship between picks and querieswill be assigned. Alternatively or additionally, an interruption ofsufficient duration can indicate a break in sessions. A search logexcerpt, in accordance with one embodiment of the invention, is shownbelow as Table 2. In various alternative embodiments, any other itemscould be captured in the search log, but are excluded here for clarity.

TABLE 2 Row Timestamp User ID Query Pick (URL) 101 Jan. 1, 2003 00:00:00U1 Q1 P5 102 Jan. 1, 2003 00:01:00 Q2 P1 103 Jan. 1, 2003 00:02:00 P2104 Jan. 1, 2003 00:02:05 P3 201 Jan. 2, 2003 00:00:00 U2 Q2 P4 202 Jan.2, 2003 00:01:00 P1 203 Jan. 2, 2003 00:02:00 P4 204 Jan. 2, 200300:04:00 Q1 P2 205 Jan. 2, 2003 00:04:05 P3 301 Jan. 3, 2003 00:00:00 U3Q3 P3 302 Jan. 3, 2003 00:04:00 Q2 P1 303 Jan. 3, 2003 02:00:00 Q3 P5401 Jan. 4, 2003 00:00:00 U2 Q1 * 402 Jan. 4, 2003 00:06:00 Q2 P4 (*=query with no associated pick)

FIG. 4 illustrates the search log for each of three independent users inaccordance with one embodiment of the invention. Search log excerpt 400shown in FIG. 4 contains search logs 410, 420, and 430 depicting thesearch information for three independent users, U1, U2, and U3,respectively, as described above in reference to Table 2. The dashedboxes in each search log represent the Q2RP portion of the Q2P searchinformation. For example, search log 410 includes Q2RP portion 411 inwhich a query Q1 has resulted in a pick P5. Search log 410 also includesQ2RP portion 412 in which a query Q2 has resulted in picks P1, P2, andP3.

Search log 420 includes an interruption 421 of 48 hours duration. Forone embodiment of the invention, such a lengthy interruption maydelineate two separate sessions across which no relationship betweenpicks and queries will be assigned. In contrast, search log 430 includesan interruption 431 of 2 hours duration. For one embodiment of theinvention such interruption may not delineate two separate usersessions. That is, all of the search activity of search log 430 may beviewed as a single user session and the information correlatedaccordingly.

Table 2A illustrates a tabulation of the click information contained inTable 2 in accordance with an embodiment of the invention. Forcomparison, Table 2B illustrates a tabulation of the click informationcontained in Table 2 in accordance with a typical prior art

TABLE 2A (Q2P Results) Q1 Q2 Q3 P1 2 3 1 P2 1 1 — P3 2 3 1 P4 1 1 — P5 12 1

TABLE 2B (Q2RP Results of Prior Art) Q1 Q2 Q3 P1 — 3 — P2 1 — — P3 1 1 1P4 — 2 — P5 1 — 1

Due to the fact that numerous factors can vary or penalize the scores,we will assume 1 pick=a score increment of +1, except for the followingpenalization situations, where we will assume the pick represents ascore increment of 0. Assuming a time threshold, the click in row 103 ispenalized in both tabulations due to the user spending a very short timeat the URL. Assuming daily database batch updates, the click in row 203would typically be penalized by the prior art tabulation of Table 2B asa duplicate of click 201. The clicks in rows 203 and 402 are penalizedby the tabulation, in accordance with an embodiment of the invention, asduplicates of click 201.

For Query Q1, URL P1, which was never clicked immediately subsequent toQ1, has garnered a high score in the tabulation, in accordance with anembodiment of the invention, because multiple users chose it before orafter—though not immediately after—issuing Query Q1. The whole matrix ofscores for the tabulation, in accordance with an embodiment of theinvention, is richer, as many more associations are noted. Some scores,such as that for Q2P4, are lower, due to the retention of session dataindicating that all the clicks came from a single user, permitting theidentification of more duplicates.

In practical applications of Q2P, we can retain the distinction as towhether a particular association was Q2RP or non-Q2RP. A single,uncorrelated non-Q2RP click (such as Q3P1 in the table) may not inspireenough confidence to release the result to users, whereas for a single,uncorrelated Q2RP click, the association is reinforced by the fact thatthe search engine presented the result for the original search.

Pick-to-Query

A pick-to-query (P2Q) correlation associates all queries recorded duringa user session that are correlated with a given pick issued during thatuser session. The search log excerpt of Table 2 illustrates the outputof P2Q correlation. That is, the same data generated for Q2P can bere-indexed for P2Q

Query-to-Query

A query-to-query (Q2Q) correlation associates all queries issued duringa user session with all other queries issued during that session. Forone embodiment, a score may be assigned to each association based uponvarious factors, including the time between queries, the number ofintervening queries and/or picks, age or order of the association(relative to older or newer associations), whether or not the queryresults generated picks, and the pair-wise order of the associatedqueries, among others.

Determining if the query results generated picks, as well as thepair-wise order of the associated queries, can be particularlyinformative, as they can indicate whether one query is a “correction” ofanother. For any practical application, it is useful to know which oftwo associated queries is an error, and which, a correction.

A search log excerpt, in accordance with one embodiment of theinvention, is shown below as Table 3. Only the query portion of thesearch log is required to create a Q2Q table.

TABLE 3 Row Timestamp User ID Query 101 Jan. 1, 2003 00:00:00 U1 Q1 102Jan. 1, 2003 00:01:00 Q2 103 Jan. 1, 2003 00:02:00 104 Jan. 1, 200300:02:05 201 Jan. 2, 2003 00:00:00 U2 Q2 202 Jan. 2, 2003 00:01:00 203Jan. 2, 2003 00:02:00 204 Jan. 2, 2003 00:04:00 Q1 205 Jan. 2, 200300:04:05 301 Jan. 3, 2003 00:00:00 U3 Q3 302 Jan. 3, 2003 00:04:00 Q2303 Jan. 3, 2003 02:00:00 Q3 401 Jan. 4, 2003 00:00:00 U2 Q1 402 Jan. 4,2003 00:06:00 Q2

Table 3A illustrates a tabulation of the click information contained inTable 3 in accordance with an embodiment of the invention (assuming theorder of queries issued is ignored).

TABLE 3A (Q2Q Results) Q1 Q2 Q3 Q1 — 2 — Q2 — — 1 Q3 — — —

The lower triangular area of Table 3A can be used to retain thepair-wise query order information, avoiding double-booking cases likerows 301–303.

As noted above, a scoring scheme may be employed in which numerousfactors can vary or penalize the score. For example, duplicates (e.g.,association in rows 101 and 102 and associations made in rows 401 and402) could be penalized. Or, for example, an uncorrelated Q2Qassociation, like Q2Q3, would not inspire enough confidence to releasethe result to users.

Pick-to-Pick

A pick-to-pick (P2P) correlation associates all picks issued during auser session with all other picks issued during that session, and assuch, is analogous to the Q2Q correlation described above. Again, inaccordance with various embodiments, a score may be assigned to eachassociation based upon various factors, including the time betweenpicks, the number of intervening queries and/or picks, age or order ofthe association (relative to older or newer associations), and thepair-wise order of the associated picks, among others.

A search log excerpt, in accordance with one embodiment of theinvention, is shown below as Table 4. Only the pick portion of thesearch log is required to create a P2P table.

TABLE 4 Row Timestamp User ID Pick (URL) 101 Jan. 1, 2003 00:00:00 U1 P5102 Jan. 1, 2003 00:01:00 P1 103 Jan. 1, 2003 00:02:00 P2 104 Jan. 1,2003 00:02:05 P3 201 Jan. 2, 2003 00:00:00 U2 P4 202 Jan. 2, 200300:01:00 P1 203 Jan. 2, 2003 00:02:00 P4 204 Jan. 2, 2003 00:04:00 P2205 Jan. 2, 2003 00:04:05 P3 301 Jan. 3, 2003 00:00:00 U3 P3 302 Jan. 3,2003 00:04:00 P1 303 Jan. 3, 2003 02:00:00 P5 401 Jan. 4, 2003 00:00:00U2 * 402 Jan. 4, 2003 00:06:00 P4

Table 4A illustrates a tabulation of the click information contained inTable 4 in accordance with an embodiment of the invention (assuming theorder of picks recorded is ignored).

TABLE 4A (P2P Results) P1 P2 P3 P4 P5 P1 — 1 3 1 2 P2 — — 1 1 — P3 — — —1 2 P4 — P5

Again, duplicates (associations involving row 201 and those involvingrow 203) could be penalized, as well as the user spending a very shorttime at the URL.

The lower triangular area of Table 4A can be used to retain thepair-wise pick order information, avoiding double-booking cases likerows 201–203.

Combinations of Exemplary Use Activity Correlations

In accordance with various alternative embodiments of the invention, twoor more correlations (e.g., the basic correlations described above) canbe concatenated to provide more relevant search results. For example,two or more basic correlations may be concatenated so as to mock thebasic correlations in order to augment their results, especially incases of sparse data, or generate additional products requiring broadmatching.

Concatenated correlations use picks or queries, rather than users, toform the links between other picks and queries. In general, therefore,the more correlations concatenated, the farther afield from the initialpick or query the results will be. Thus, in many cases, concatenatingthe fewest correlations to effect the desired result will be the optimalapproach.

Table 5 illustrates the possible combinations of two correlations forthe basic correlations of USE activity described above.

TABLE 5 Basic Correlation Combination analogy Yields QPQ Q2P + P2Q Q2QQueries related to picks related to the original query QQQ Q2Q + Q2Q Q2QQueries related to queries related to the original query PQQ P2Q + Q2QP2Q Queries related to queries related to the original pick PPQ P2P +P2Q P2Q Queries related to picks related to the original pick PPP P2P +P2P P2P Picks related to picks related to the original pick PQP P2Q +Q2P P2P Picks related to queries related to the original pick QQP Q2Q +Q2P Q2P Picks related to queries related to the original query QPP Q2P +P2P Q2P Picks related to picks related to the original query

Concatenating correlations can introduce erroneous relationships;therefore, for one embodiment of the invention, the correlations arecorrelated. For example, if a Q2Q correlation requires two independentusers, a QQQ correlation string should require two queries that link theoriginal and final queries. In QQQ, a mutually associated query (Q3)associates a query (Q1) with another query (Q2). If two or moreindependent, mutually associated queries make the same association, thatis a correlation.

As discussed above in reference to basic correlations, every associationbetween the original pick or query and the output pick or query, wasmade by at least two individuals (or by a search engine plus oneindividual). With concatenated correlations, it is possible that noindividual user (or search engine) has ever associated the original pickor query with any of output picks or queries. An indirect correlationwould still require a minimum of two independent users.

In effect, concatenated correlations tend to predict what the basiccorrelations might look like at some future point in time where muchmore data has been collected. Essentially, they identify likelycorrelations that simply have not yet been observed.

Combinations of correlations require correlation through multipleintermediate picks or queries. The number of distinct picks and/orqueries linking the original and related picks or queries is moreimportant than how closely the intermediate picks and/or queries arelinked to the original and related picks and/or queries. For acorrelation, there must be at least two distinct link paths, regardlessof how many users have established those links. That is, associating Q1and Q2, via one intermediate node Q3, even if multiple users haveestablished that association, does not constitute a correlation betweenQ1 and Q2.

A number of factors contribute to the strength of a correlation,including the number of link paths, the strength of each directcorrelation component link, and the distinctiveness of each intermediatenode. For example, linking two picks that are correlated through acommon and general query (e.g., “cars”) will likely result in a muchweaker correlation than linking two picks via a more distinctiveintermediate query like (e.g., “1965 ford mustang convertible”).

The advantages of concatenated correlations may be better discerned bythe following example of the concatenated correlation Query to Pick toQuery (QPQ).

Table 6A and Table 6B illustrate the results of multiplying theintersecting QP scores of Tables 2A and 2B, respectively, and summingthe results over the queries to determine the combined associationscores. (This is not necessarily an optimal algorithm, but is used forexemplary purposes).

TABLE 6A P1 P2 P3 P4 P5 Sum Q1–Q2 2 × 3 1 × 1 2 × 3 1 × 1 1 × 2 16 Q1 toQ3 2 × 1 — 2 × 1 1 × 1 5 Q2 to Q3 3 × 1 — 3 × 1 2 × 1 8

TABLE 6B P3 P5 Sum Q1 to Q2 1 × 1 — 1 Q1 to Q3 1 × 1 1 × 1 2 Q2 to Q3 1× 1 — 1

Table 6C and Table 6D illustrate the equivalent Q2Q correlation tablesfor Table 6A and Table 6B, respectively.

TABLE 6C Q1 Q2 Q3 Q1 — 16 5 Q2 — 8 Q3 —

TABLE 6D Q1 Q2 Q3 Q1 — 1 2 Q2 — — 1 Q3 — — —

As shown, the QPQ correlation results, in accordance with one embodimentof the invention, generate considerably more correlations than thenarrower QRPQ correlation of a more specific alternative embodiment ofthe invention, and considerably more than the Q2Q correlation inaccordance with still another alternative embodiment of the invention.Furthermore, using the QPQ correlation allows for providing suggestionsfor poorly formed queries that do not generate search results. This wasnot possible with the prior art schemes.

User-to-User

Just as queries and/or picks can be correlated by users, users can becorrelated by queries and/or picks. The general process, which is anindirect correlation analogous to a QPQ correlation, will be referred toas a user-to-user (U2U). A U2U correlation, based upon result picks(i.e., two users have entered the same query and pick), is illustratedbelow in Table 7. Such a correlation would be a user-to-resultpick-to-user (URPU), though there are other U2U correlations inaccordance with alternative embodiments of the invention. Again, inaccordance with various embodiments, a score may be assigned to eachassociation based upon various factors. For example, assuming a timethreshold, the click in row 103 is penalized due to the user spending avery short time at the URL.

TABLE 7 Row Timestamp User ID Query Pick (URL) 101 Jan. 1, 2003 00:00:00U1 Q1 P5 102 Jan. 1, 2003 00:01:00 Q2 P1 103 Jan. 1, 2003 00:02:00 P2104 Jan. 1, 2003 00:02:05 P3 201 Jan. 2, 2003 00:00:00 U2 Q2 P4 202 Jan.2, 2003 00:01:00 P1 203 Jan. 2, 2003 00:02:00 P4 204 Jan. 2, 200300:04:00 Q1 P2 205 Jan. 2, 2003 00:04:05 P3 301 Jan. 3, 2003 00:00:00 U3Q3 P3 302 Jan. 3, 2003 00:04:00 Q2 P1 303 Jan. 3, 2003 02:00:00 Q3 P5401 Jan. 4, 2003 00:00:00 U2 Q1 * 402 Jan. 4, 2003 00:06:00 Q2 P4

Table 7A illustrates a tabulation of the click information contained inTable 7 in accordance with an embodiment of the invention.

TABLE 7A U1 U2 U3 Q1P2 — 1 — Q1P3 — 1 — Q1P5 1 — — Q2P1 1 1 1 Q2P2 Q2P31 1 Q2P4 1 Q3P3 1 Q3P5 1

In accordance with one embodiment of the invention, the affinity of agiven user (e.g., U1) toward another user (e.g., U2) may be defined asthe number of queries/picks the user shares with another, divided by thetotal number of queries/picks of the given user (i.e.,Affinity_(U1-U2)=(shared QP_(U1-U2))/(total QP_(U1)). For alternativeembodiments of the invention, more sophisticated affinity algorithms maybe used. For example, an affinity algorithm, in accordance with oneembodiment, might account for differences in frequency of search andaward different weights to queries, picks, and query-picks.

Table 7B illustrates the calculated affinity information for thetabulated click information of Table 7A.

TABLE 7B To U1 To U2 To U3 U1 1 0.67 0.33 U2 0.40 1 0.20 U3 0.33 0.33 1

Using such affinity information, when subsequently generating resultsfor a future query issued by a given user, the pick scores forcorresponding users are adjusted according to the affinity information.For example, results for a future query of user U1, would adjust scoresfor picks generated by user U2 by a function of (0.67) and scores forpicks generated by user U3 by a function of (0.33). Picks generated by auser that had no affinity would be assigned some default value.

Exemplary Applications

The methods and systems for organizing and presenting data discussedabove, in reference to various alternative embodiments of the invention,may be used for many practical applications as will be obvious to thoseskilled in the art. A number of such applications will be discussed,more fully, below. The use of particular USE activity information and/oruser information may be more suitable to particular applications thanothers. For example, for a particular application, it may be morepractical, more efficient, or more accurate, to correlate particular USEactivity. The following exemplary applications will be discussed interms of the USE activity information and/or user information that iswell suited to a practical implementation of the particular application.

Exemplary Q2P Applications

Embodiments of the invention that correlate the Q2P and/orQ2P-equivalent combination USE activity information allow a user toobtain more relevant search results in a number of ways. For example, auser may refine a search and associate some portion of the revisedresults with the original search. That is, picked documents need nothave a textual relationship to the original search, but only aconceptual relationship. The conceptual relationship may provide abetter response to the original search. Search results based upon acorrelation of Q2P USE activity information circumvents the limitationof text-based retrieval found in the prior art schemes. Embodiments ofthe invention have the capability to preserve and exploit the userresearch process. This capability can be used to effect systems with anumber of distinct advantages over prior art schemes.

The correlation of Q2P USE activity information for a number ofindependent users, in accordance with various embodiments of theinvention, takes advantage not only of the relevance judgments ofprevious users, but of their research efforts as well. Subsequent usersneed not reiterate the mistakes of previous users; instead, subsequentusers benefit from the trial-and-error learning of previous users.

FIG. 5 illustrates a process for providing more relevant search resultsin accordance with one embodiment of the invention. Process 500, shownin FIG. 5, begins at operation 505 in which a query is received from auser. Such a query may have one or more particular characteristics,which, once discerned, may be the basis for providing more relevantsearch results in accordance with various alternative embodiments of theinvention.

At operation 510, the Q2P USE activity information for a number of usersis correlated in regard to the received query. Each query may havevarious particular characteristics that may be determined throughcorrelation of Q2P USE activity information. These particularcharacteristics may include, for example, the following: the query maybe subject to different meanings for different users, the query may bemisspelled, the query may have equivalent phrasings, the query may havemore relevant portions and less relevant portions, the query may beassociated with a specific result product or result product mix, and thequery may be subject to broader or narrower search results. Each ofthese characteristics, as well as others, alone or in combination, maybe useful in providing more relevant search results under differentcircumstances.

At operation 515, search results, based upon the correlated Q2P USEactivity information, are provided to the user in response to the query.The search results provided may be based upon one or morecharacteristics of the query. For example, where it is determined thatthe query has different meanings, results based upon a more popularmeaning may be provided in response to the query.

Several examples of results obtained by process 500 in comparison toresults obtained through typical prior art methods are shown below inrelation to the particular characteristics of queries discussed above.

Table 8 illustrates exemplary search results for several queries havingmore than one meaning. As shown, embodiments of the invention allow thesearch results to relate to the more popular meaning of the particularquery in comparison to a typical prior art scheme.

TABLE 8 Dominant interpretation Example query Exemplary Embodiment PriorArt Scheme “buffalo” Most users seek the All the top results animal (andoften enter concern Buffalo, NY. the subsequent query “bison” in orderto find more relevant results.) “holes” Most users seek the book All thetop results or recent movie Holes. concern “black holes.” “windows” Alarge fraction of users All the top results seek glass windows. concernMicrosoft Windows. “parties” A large fraction of users All the topresults seek party planning or concern political parties. supplies.

Table 9 illustrates exemplary search results for a misspelled query(e.g., “encycopidea”). As shown, embodiments of the invention allow thesearch results to relate to the probable correctly spelled query. Inthis way, embodiments of the invention can determine the correctspelling for a misspelled query. Prior art schemes typically produce nosearch results, or poor search results, for such misspellings.

TABLE 9 Dominant interpretation Example query Exemplary Embodiment Priorart scheme “encycopidea” 1. www.brittanica.com Issues no results, just a2. www.encyclopedia.com spelling suggestion to try “Encyclopedia.”

For one embodiment of the invention, such a spelling correction is a“soft” correction. That is, in accordance with one embodiment of theinvention, the results provided in response to the query are the resultsthat were preferred by a majority of the users who entered the exactquery. If most users consider the query a misspelling, then many resultswill contain the corrected query. If most users consider the queryintentional, then most results will contain the query unchanged. If bothinterpretations are legitimate, the results may be a combination. Forsuch embodiments, it becomes highly unlikely that an erroneous spellingcorrection will be provided because all corrections are conceptuallyrelevant, not just textually similar.

In contrast, prior art spelling corrections are typically “hard”corrections. That is, such schemes identify a misspelled query, attemptto correct it and then search for results based upon the correction.Such schemes provide irrelevant results where legitimate queries aremisdiagnosed as misspelled, or if the query is in fact misspelled, butthe algorithmic correction was also not the intended query. An “active”correction would require a user to click a link to search again on asuggested query, which again might be an erroneous correction.

Table 10 illustrates exemplary search results for a query having two ormore representations or equivalent phrasings (e.g., “Burma andMyanmar”). As shown, embodiments of the invention provide search resultsfor each equivalent phrasing with a high degree of commonality. For theexample provided, five of the top nine search result URLs are common toeach of the equivalent phrasings. This is much higher similarity thanprior art schemes.

TABLE 10 Exemplary Embodiment Example query: “burma history” Examplequery: “myanmar history” 1. www.travel- 1. www.hartford-   burma.com/history.html    hwp.com/archives/54/index- 2. www.hartford-   b.html    hwp.com/archives/54/index- 2. www.myanmars.net/history/   b.html 3. www.travel- 3. www.asiatour.com/myanmar/e-   burma.com/history.html    01land/em-lan43.htm 4.rubens.anu.edu.au/student.projects/ 4. rubens.anu.edu.au/   offerings/burma.2.html    student.projects/offerings/ 5.www.asiatour.com/myanmar/e-    burma.2.html    01land/em-lan46.htm 5.www.nationbynation.com/ 6. www.myanmar.com/    Burma/History1.html 7.www.odci.gov/cia/publications/ 6. www.louisville.edu/library/   factbook/geos/bm.html    ekstrom/govpubs/ 8. www.myanmars.net/   international/burma/ 9. www.asiatour.com/myanmar/e-    burmahistory.html   01land/em-lan43.htm 7. www.myanmars.net/ 8. asiatours.net/burma/info/   history.html 9. pilger.carlton.com/burma

Users often include keywords that add little relevant information to aquery, yet impose unnecessary text-matching requirements on the searchengine that result in less relevant search results for prior artschemes. Embodiments of the invention overcome this disadvantage.

Table 11 illustrates exemplary search results for a query including aredundant portion. As shown, embodiments of the invention allowredundant, non-essential, and non-relevant portions of the query to beignored, where doing so provides a more relevant, simpler query.

TABLE 11 Rank of expected #1 result Exemplary Prior Example queryEmbodiment Art Scheme “columbus blue jackets” 1 1 “columbus ohio bluejackets” 1 Not in top 100

In accordance with one embodiment of the invention, search result pickson various result products (e.g., graphics, audio/video, text, images,news items, etc.) are treated equally. That is, users need not specifywhich type of result they are seeking. Results issued can reflect priorpreferences exhibited by the user or the preferences of previousindependent users. For example, if a query precipitates an image searchso frequently that a particular image is the highest scoring pick, thatimage may be provided as a search result in accordance with its score.For one embodiment, the various top-scoring results of different resultproducts need not be interleaved in score order, but may be grouped byproduct for presentation. For one such embodiment, the result productsthe user seeks are identified without the need to decipher the querywith language tools in an attempt to ascertain the user's objectives.

Table 12 ilustrates exemplary search results for a query resulting in asearch result product mix (e.g., including graphics results). As shown,embodiments of the invention allow search results that include a productmix.

TABLE 12 Example query Exemplary Embodiment search results “who ishomer 1. www.simpsons100.com simpson?” 2. www.thesimpsons.com 3.www.snpp.com 4. members.aol.com/jynelms/tv/simpsons.html 5.

6. www.newspringfield.com

Embodiments of the invention provide the ability to vary the breadth ofthe search results by adjusting the weighting of the Q2P vs. Q2RP pickscores. Table 13 illustrates narrow and broad search results for thequery. “stanford”.

TABLE 13 Example query Q2RP/Q2P = 1 Q2RP/Q2P = 0 “stanford”www.stanford.edu www.harvard.edu www-cs.stanford.edu www.yale.eduwww.gostanford.com www.berkeley.edu www-med.stanford.eduwww.princeton.edu www-sul.stanford.edu www.ucla.edu

Column 1 of Table 13 contains narrow results related to picks on thequery. Column 2 of Table 13 shows broad results (e.g., only picks thatwere not made in response to the results of the original query areshown). The column 2 results represent the broad scope of the querytopic, whereas the column 1 results help the user explore the depth ofthe query topic. Practically, a combination of the narrow and broadresults may provide the most relevant search results. For one embodimentof the invention, a number of combinations having varying breadths arecreated, allowing the user to select a “broaden the results” or “focusthe results” link, or vary a control to adjust the mixture.

Exemplary Q2Q Applications

One embodiment of the invention that correlates the Q2Q and/orQ2Q-equivalent combination USE activity information allows users toobtain suggested queries relevant to their search. FIG. 6 illustrates aprocess for providing relevant query suggestions in accordance with oneembodiment of the invention. Process 600, shown in FIG. 6, begins atoperation 605 in which a query is received from a user.

At operation 610, the Q2Q (and/or Q2Q-equivalent combination) USEactivity information is correlated. Correlating the Q2Q USE activityinformation directly tends to produce results that are predominantlyrefinements of the query. Correlating Q2Q-equivalent combination USEactivity information tends to produce results that are more diverse,although there is typically a high degree of overlap. For one embodimentof the invention, Q2Q-equivalent combination USE activity information iscorrelated for a relative obscure original query, because such acorrelation often produces considerably more query suggestions.

Table 14 illustrates query suggestions provided in response to theoriginal query, “electronic eavesdropping devices,” in accordance withone embodiment of the invention. Column 1 of Table 14 contains querysuggestions based upon the correlation of Q2Q USE activity informationdirectly, while column 2 of Table 14 contains query suggestions basedupon the correlation of Q2Q equivalent combination (i.e., QPQ) USEactivity information.

TABLE 14 Example query Direct (Q2Q) Indirect (QPQ) “electronicelectronic eavesdropping electronic eavesdropping eavesdropping what iselectronic eavesdropping devices devices” eavesdropping devices?eavesdropping eavesdropping technology why do we need electroniceavesdropping? where can I find detective devices? free keystrokerecorders electronic surveillance countermeasures spystuff fmeavesdropping devices

At operation 615, one or more query suggestions, based upon thecorrelated Q2Q (and/or Q2Q-equivalent combination) USE activityinformation, are provided to the user. For one embodiment of theinvention, query suggestions may be presented on the result page.Alternatively or additionally, a link may be provided to a page of querysuggestions. Such an embodiment may be practical where an original queryresults in a large number of highly relevant (e.g., high scoring) querysuggestions. For one embodiment of the invention, query suggestions canbe sorted into refinements (containing all the original search terms)and related searches, prior to presentation.

In accordance with one embodiment of the invention, correlation of Q2Q(and/or Q2Q-equivalent combination) USE activity information, is used toeffect an active query spelling correction scheme. An active queryspelling correction scheme is one in which the user selects a suggestedquery correction to obtain search results based upon the querycorrection.

FIG. 6A illustrates a process for providing query spelling correctionsuggestions in accordance with one embodiment of the invention. Process600A, shown in FIG. 6A, begins at operation 605A in which a query isreceived from a user. Such a query may be a misspelling of the user'sintended query.

At operation 610A, the Q2Q (and/or Q2Q-equivalent combination) USEactivity information is correlated, as discussed above, in reference tooperation 610 of process 600.

At operation 611A, one or more query suggestions are determined basedupon the correlated Q2Q (and/or Q2Q-equivalent combination) USE activityinformation.

At operation 612A, one or more of the query suggestions are determinedas spelling corrections of the original query (i.e., the query receivedat operation 605A). In accordance with alternative embodiments of theinvention, the determination of the query suggestion is effected invarious different manners depending upon the query received. Forexample, for previously observed queries, where Q2Q information exists,conceptually related queries that are also textually similar can beidentified using the Q2Q correlation information and an edit distancecomputation algorithm. In such a case, various criteria may be used toeffect the determination of a query suggestion as a spelling correctionof the original (i.e., received) query. For example, where thedetermined query suggestion is correlated to the received query,textually similar to the received query, and more common than thereceived query, the determined query suggestion may be determined as aspelling correction of the received query. For alternative embodiments,more criteria may be considered to bolster confidence in thedetermination of the query suggestion as a spelling correction of thereceived query. For example, where the query suggestion tends to beissued subsequent to the received query more often than prior to thereceived query or where the query suggestion tends to result in moreuser picks than the received query, these criteria would bolsterconfidence in the determination of the query suggestion as a spellingcorrection of the received query.

Where the query received at operation 605A is a previously unobservedquery, then no Q2Q information exists. In such cases, in accordance withone embodiment of the invention, if a word in a query is suspected ofbeing an error, it is evaluated in one of two ways depending uponwhether or not the suspect word has been previously observed or not.

Where the suspect word has been previously observed, then other queriesin which the word appears are identified. These queries are weightedbased on keywords in common with the received query. Finally, previouslyconsidered corrections of the suspect word are examined and used toprovide a suggested correction based on the frequency of such previouslysuggested corrections and keyword weighting of the queries in which thesuspect word appears.

Where the suspect word has not been previously observed, other queriesin which all the other keywords in the received query appear can beidentified. For one such embodiment of the invention, where there are noqueries that meet this criteria, the queries with the most distinctive(low frequency) words in common with the received query can beidentified. For either embodiment, the identified queries can beweighted based on textual similarity with the received query and themost common query with high textual similarity to the received queryidentified.

For one embodiment of the invention, where conditions do not permit useof the method spelling correction suggestion process described above,the correlated Q2Q USE activity information is used in conjunction witha conventional n-gram-type model. In such an embodiment, wordassociation frequencies are extracted from the query frequency data todetermine the common partners for known keywords or phrases. This data,in combination with edit distance, is used to determine a query as alikely spelling correction suggestion for an unknown word in amulti-word query.

At operation 615A, one or more query suggestions that have beendetermined as a spelling correction of the received query, are providedto the user as likely spelling corrections of the received query. Forone embodiment of the invention, where no query suggestion is determinedas a likely spelling correction of the received query, the querysuggestions are presented as described above in operation 615 of process600 in reference to FIG. 6.

Exemplary P2Q Application

In accordance with one embodiment of the invention, suggested queriesrelated to a particular result page can be provided using P2Q or aP2Q-equivalent combination.

FIG. 7 illustrates a process in which suggested queries are provided inaccordance with one embodiment of the invention. Process 700, shown inFIG. 7, begins at operation 705 in which a query and correspondingsearch result, providing a number of result URLs, is received.

At operation 710, the P2Q (or P2Q-equivalent combination) USE activityinformation is correlated for each result URL.

At operation 715, suggested queries are provided for each result URLbased upon the correlated P2Q USE activity information. That is, a listof suggested queries that are closely related to any of the pages amongthe search results is provided. Such suggestions would, in the case ofambiguous queries, provide a user with a focused meaning that may beembodied by one of the URLs, but not others.

Table 15 illustrates an exemplary search result page for the query“rangers” and exemplary corresponding suggested queries in accordancewith one embodiment of the invention.

TABLE 15 Result page Suggested queries www.rangers.co.uk glasgow rangersrangers football club rangers football www.specialoperations.com armydelta force special forces airborne rangers www.newyorkrangers.com newyork rangers rangers logo where can I find info on the rangers?www.ford-trucks.com 1987 ford ranger ford truck picturesrangers.mlb.com/NASApp/mlb/tex/homepage/ mlb rangers tex_homepage.jsptexas rangers baseball texas rangersExemplary P2P Applications

In accordance with one embodiment of the invention, one or more resultsthat are similar to a particular result, provided in response to areceived query, are provided using P2P or a P2P-equivalent combination.For example, a query is received and a result corresponding to the queryis evaluated. Based upon the evaluation, similar results are alsoprovided. That is, for example, a few related pages could be specifiedalong with the result and/or a link could lead to a new result page withadditional similar results. Typically, similar results for an image willyield mostly other images, similar results for a web page will yieldmostly other web pages, etc.

FIG. 8 illustrates a process in which similar results are provided inaccordance with one embodiment of the invention. Process 800, shown inFIG. 8, begins at operation 805 in which a query is received and acorresponding search result determined.

At operation 810, the P2P (or P2P-equivalent combination) USE activityinformation is correlated for the corresponding search result.

At operation 815, one or more similar results (i.e., similar to thereceived result) are provided based upon the correlated P2P USE activityinformation. That is, a list of results that are closely related to thereceived result is provided. Similar results can constitute searchpicks, images, news items, etc.

Table 16 illustrates exemplary similar results for a result provided inresponse to the query “phoenix” in accordance with one embodiment of theinvention.

TABLE 16 Result Top similar results www.ci.phoenix.az.uswww.azcentral.com www.phoenixcvb.com www.arizonaguide.comwww.go-arizona.com/phoenix www.state.az.us

 

 

Exemplary User Information Applications Personalized Search

The concept of personalized searches is based on the premise that morerelevant search results can be provided if something is known about theuser. The recommendations of users who have shared at least some of aparticular user's interests and opinions in the past are deemed to be ofgreater value than the recommendations of users, who have never shownevidence of sharing his interests and tastes.

Prior art personalized search schemes typically identify the demographicof a user, then tailor the results to the preferences of other membersof the demographic community. This approach has serious drawbacks inthat preferences vary widely within a demographic community. Each useris typically a member of many, often difficult to reconcile, demographiccommunities, and users are typically not forthcoming with reliabledemographic information.

In accordance with one embodiment of the invention, each user is viewedas a community of one, with a degree of affinity to other users.

FIG. 9 illustrates a process in which U2U information is correlated toprovide more relevant search results in accordance with one embodimentof the invention. Process 900, shown in FIG. 9, begins at operation 905in which an affinity matrix is calculated for every user whodemonstrates at least a minimum amount of search activity. Calculationof such a matrix could proceed as follows for one embodiment of theinvention. First, all queries and picks by a given user, U1, areextracted. Then users, U2, that duplicated at least N of U1's queriesand/or picks are identified, and U2's remaining queries and picks areidentified. Then weights are assigned to each query and pick and anaffinity score calculated. The weights may be inversely proportional tohow common among all users that query and/or pick was, and the affinityscore is calculated from a comparison of the weighted picks/queries incommon with the total weighted picks/queries.

At operation 910, the search results, resulting from a new query of U1,are analyzed, and the score of each result is increased if any previouspicks were due to users with a non-zero affinity score.

At operation 915, the results are reordered based upon the scoreadjustments of operation 910 and the reordered results are presented tothe user.

In accordance with an alternative embodiment, a process similar toprocess 900 can be used to increase the relevancy of search suggestions.

Localized Search

One important type of user information is location. Location has theadvantage over other demographic information, in that it does notrequire self-reporting by the user and holds true no matter who is usingthe computer.

For one embodiment of the invention, user information includesgeo-location information for users and, by extension, their queries andpicks. The geo-location information may include latitude and longitudeas well as city, state and country name. In accordance with oneembodiment, such user information is used to provide search resultsbased upon a user's geographic location. For example, a US user enteringthe query “osu” might reasonably mean “Ohio State University”, “OklahomaState University” or “Oregon State University.” The search resultsprovided are based upon user location (e.g., a user in Oregon isprovided with “Oregon State University”). For one embodiment, the searchresults provided are also correlated with users having similar userinformation (e.g., similar location).

FIG. 10 illustrates a process by which more relevant search results areprovided to a user based upon the user's location in relation to alocation associated with a pick in accordance with one embodiment of theinvention. Process 1000, shown in FIG. 10, begins at operation 1005 inwhich a URL that has acquired more than a specified number of picks isassigned a latitude-longitude “centroid” and a “sphere of influence.”The centroid is the position where some function of the sum of thedistances to each of the statistically significant logged user-picks isminimized. For example, for one embodiment, the centroid will representthe location of the actual brick-and-mortar incarnation of the URL (theURL exhibiting local bias) to a high degree of accuracy. Where no localbias exists, the centroid location has little or no meaning. Forexample, a URL without a brick-and-mortar incarnation may exhibit nolocal bias on the national scale.

The sphere of influence is a geographic radius within which the URL isexpected to be highly appealing to users and outside of which it isexpected to be largely unappealing. The smaller the radius, the steeperthe drop-off in appeal—or the more “localized” the appeal. The radius ofthe sphere of influence is inversely proportional to some measure oflocal bias. For example, a highly localized URL likewww.canariesbaseball.com forms a very tight sphere of influence centeredon Sioux Falls, S. Dak. (i.e., the hometown of the Sioux FallsCanaries).

At operation 1010, the sphere of influence is adjusted based uponstatistical data. For example, some locations generate many more clicksthan other locations. Raw click score data is adjusted to reflect thedisparity.

At operation 1015, the user's geographic location is compared with a setof pre-computed location data for responses to a particular query andthe search result scores are adjusted based upon the geographic locationof the user. The scores of those URLs that are nearby and have a highdegree of localization are increased, while the scores of distant URLsare decreased (scores of URLs without much localization bias are leftunchanged).

At operation 1020, the user is provided with search results that arebased, at least in-part, upon the geographic location of the user. Thiswill allow users to receive subjectively relevant results within theinitial several results. For example, in contrast to prior art schemes,a user in Wyoming who enters “state tax forms” will be provided withstate tax forms for Wyoming more readily.

Localization by User

Localization can be based on picks, as was described above, or can bebased on the user in accordance with alternative embodiments. For oneembodiment of the invention, localization based upon the user allowsresults for popular queries to be cached by jurisdiction and then issuedto users based on location without any real-time calculations.

FIG. 11 illustrates a process by which more relevant search results areprovided to a user based upon the user's location in accordance with oneembodiment of the invention. Process 1100, shown in FIG. 11, begins atoperation 1105 in which jurisdictions that account for a specifiedportion of the clicks for a given query are determined, starting withthe highest-level jurisdictions. For example, the query “car insurance”yields a mix of US and UK sites, with a majority of clicks on the UKsites from users located in the UK and a majority of clicks on the USsites from users located in the US.

At operation 1110, the click score for jurisdictions accounting for aspecified threshold portion of the clicks is increased, and a separateresult list, tagged to that jurisdiction, is created.

At operation 1115, the click scores for users outside of thatjurisdiction are decreased, creating a general list for users outsidethe jurisdictions that account for a specified threshold portion of theclicks. In an alternative embodiment, real-time blended lists arecreated based on user proximity to special jurisdictions.

At operation 1116, if no jurisdictions account for a specified thresholdportion of the clicks, then the results are provided based upon thejurisdiction of the user at operation 1120.

If, at operation 1116, there still exist jurisdictions that account fora specified threshold portion of the clicks, then the process isreiterated by proceeding to lower level jurisdictions at operation 1125,and from there, to operation 1105.

For an alternative embodiment, the locations are correlated with theURLs, regardless of query. This has significant statistical advantagesbecause it aggregates more data. For example, the origins of all userswho picked a particular state tax site for any query are considered, theparticular site is then given extra weight for users in the mostprominent state regardless of the query involved. The potentialdisadvantage occurs if a Wyoming user enters “Florida tax forms.” If theWyoming state tax site is among the results, its score will get asignificant and unwarranted increase.

General Matters

Embodiments of the invention provide methods and systems forconceptually organizing and presenting information in which acorrelation of users' responses to the organization and presentation ofinformation is used to determine an optimal organization andpresentation of the information. Though described above in terms ofseveral exemplary embodiments, many additional applications exist foralternative embodiments of the invention.

The invention includes various operations. Many of the methods aredescribed in their most basic form, but operations can be added to ordeleted from any of the methods without departing from the basic scopeof the invention. The operations of the invention may be performed byhardware components or may be embodied in machine-executableinstructions as described above. Alternatively, the steps may beperformed by a combination of hardware and software. The invention maybe provided as a computer program product that may include amachine-readable medium having stored thereon instructions, which may beused to program a computer (or other electronic devices) to perform aprocess according to the invention as described above.

FIG. 12 illustrates an embodiment of a digital processing system thatmay be used for the server DPS 320, or client DPSs 305–308, as discussedabove in reference to FIG. 3, in accordance with an embodiment of theinvention. For alternative embodiments of the present invention,processing system 1201 may be a computer or a set top box that includesa processor 1203 coupled to a bus 1207. In one embodiment, memory 1205,storage 1211, display controller 1209, communications interface 1213,and input/output controller 1215 are also coupled to bus 1207.

Processing system 1201 interfaces to external systems throughcommunications interface 1213. Communications interface 1213 may includean analog modem, Integrated Services Digital Network (ISDN) modem, cablemodem, Digital Subscriber Line (DSL) modem, a T-1 line interface, a T-3line interface, an optical carrier interface (e.g. OC-3), token ringinterface, satellite transmission interface, a wireless interface orother interfaces for coupling a device to other devices. Communicationsinterface 1213 may also include a radio transceiver or wirelesstelephone signals, or the like.

For one embodiment of the present invention, communication signal 1225is received/transmitted between communications interface 1213 and thecloud 1230. In one embodiment of the present invention, a communicationsignal 1225 may be used to interface processing system 1201 with anothercomputer system, a network hub, router, or the like. In one embodimentof the present invention, communication signal 1225 is considered to bemachine readable media, which may be transmitted through wires, cables,optical fibers or through the atmosphere, or the like.

In one embodiment of the present invention, processor 1203 may be aconventional microprocessor, such as, for example, but not limited to,an Intel Pentium family microprocessor, a Motorola familymicroprocessor, or the like. Memory 1205 may be a machine-readablemedium such as dynamic random access memory (DRAM) and may includestatic random access memory (SRAM). Display controller 1209 controls, ina conventional manner, a display 1219, which in one embodiment of theinvention, may be a cathode ray tube (CRT), a liquid crystal display(LCD), an active matrix display, a television monitor, or the like. Theinput/output device 1217, coupled to input/output controller 1215, maybe a keyboard, disk drive, printer, scanner and other input and outputdevices, including a mouse, trackball, trackpad, or the like.

Storage 1211 may include machine-readable media such as, for example,but not limited to, a magnetic hard disk, a floppy disk, an opticaldisk, a smart card or another form of storage for data. In oneembodiment of the present invention, storage 1211 may include removablemedia, read-only media, readable/writable media, or the like. Some ofthe data may be written by a direct memory access process, into memory1205 during execution of software in computer system 1201. It isappreciated that software may reside in storage 1211, memory 1205, ormay be transmitted or received via modem or communications interface1213. For the purposes of the specification, the term “machine readablemedium” shall be taken to include any medium that is capable of storingdata, information or encoding a sequence of instructions for executionby processor 1203 to cause processor 1203 to perform the methodologiesof the present invention. The term “machine readable medium” shall betaken to include, but is not limited to, solid-state memories, opticaland magnetic disks, carrier wave signals, and the like.

While the invention has been described in terms of several embodiments,those skilled in the art will recognize that the invention is notlimited to the embodiments described, but can be practiced withmodification and alteration within the spirit and scope of the appendedclaims. The description is thus to be regarded as illustrative insteadof limiting.

1. A method comprising: presenting a set of conceptually relatedinformation to a plurality of users; receiving one or more responsesfrom each of two or more users, the responses in regard to apresentation of the conceptually related information; correlating theresponses from two or more users, the correlation being a pick-to-querycorrelation or a pick-to-pick correlation, wherein a pick-to-querycorrelation associates queries recorded during a user session that arecorrelated with a pick issued during the user session and a pick-to-pickcorrelation associates picks issued during the user session with otherpicks issued during the user session; and modifying the presentation ofthe conceptually related information based upon correlated responses. 2.The method of claim 1 wherein the set of conceptually relatedinformation comprises one or more distinctly formulated gathering pageshaving varying presentation.
 3. The method of claim 2 wherein the one ormore responses comprise selected and non-selected portions of the set ofconceptually related information.
 4. The method of claim 3 whereinmodifying the presentation of the conceptually related information basedupon correlated responses includes reformulating one or more of thegathering pages.
 5. The method of claim 1 wherein the responses receivedfrom each of the two or more users are search engine activityinformation of each respective user.
 6. The method of claim 5 whereinmodifying the presentation of the conceptually related informationincludes associating particular information with a corresponding userquery.
 7. The method of claim 5 wherein the search activity informationincludes all of the information of a user search engine session.
 8. Themethod of claim 5 wherein the responses received from each of the two ormore users are selected from the group of search engine activityinformation consisting of query-to-pick information, pick-to-queryinformation, pick-to-pick information, query-to-query information, andcombinations thereof.
 9. A system comprising: a server digitalprocessing system (DPS) to communicate a set of conceptually relatedinformation to a plurality of client DPSs, receive one or more userresponses from each of two or more users, the responses in regard to apresentation of the conceptually related information, correlate theresponses from two or more users, the correlation being a pick-to-querycorrelation or a pick-to-pick correlation, wherein a pick-to-querycorrelation associates queries recorded during a user session that arecorrelated with a pick issued during the user session and a pick-to-pickcorrelation associates picks issued during the user session with otherpicks issued during the user session, and modify the presentation of theconceptually related information based upon correlated responses; andone or more client DPSs to present the conceptually related informationto a plurality of users and communicate the user responses to the serverDPS.
 10. The system of claim 9 wherein the set of conceptually relatedinformation comprises one or more distinctly formulated gathering pageshaving varying presentation.
 11. The system of claim 10 wherein the oneor more responses comprise selected and non-selected portions of the setof conceptually related information.
 12. The system of claim 11 whereinmodifying the presentation of the conceptually related information basedupon correlated responses includes reformulating one or more of thegathering pages.
 13. The system of claim 9 wherein the responsesreceived from each of the two or more users are search engine activityinformation of each respective user.
 14. The system of claim 13 whereinmodifying the presentation of the conceptually related informationincludes associating particular information with a corresponding userquery.
 15. The system of claim 13 wherein the search activityinformation includes all of the information of a user search enginesession.
 16. The system of claim 13 wherein the responses received fromeach of the two or more users are selected from the group of searchengine activity information consisting of query-to-pick information,pick-to-query information, pick-to-pick information, query-to-queryinformation, and combinations thereof.
 17. A machine-readable storagemedium that provides executable instructions which, when executed by aprocessor, cause the processor to perform a method comprising:presenting a set of conceptually related information to a plurality ofusers; receiving one or more responses from each of two or more users,the responses in regard to a presentation of the conceptually relatedinformation; correlating the responses from two or more users, thecorrelation being a pick-to-query correlation or a pick-to-pickcorrelation, wherein a pick-to-query correlation associates queriesrecorded during a user session that are correlated with a pick issuedduring the user session and a pick-to-pick correlation associates picksissued during the user session with other picks issued during the usersession; and modifying the presentation of the conceptually relatedinformation based upon correlated responses.
 18. The machine-readablemedium of claim 17 wherein the set of conceptually related informationcomprises one or more distinctly formulated gathering pages havingvarying presentation.
 19. The machine-readable medium of claim 18wherein the one or more responses comprise selected and non-selectedportions of the set of conceptually related information.
 20. Themachine-readable medium of claim 19 wherein modifying the presentationof the conceptually related information based upon correlated responsesincludes reformulating one or more of the gathering pages.
 21. Themachine-readable medium of claim 17 wherein the responses received fromeach of the two or more users are search engine activity information ofeach respective user.
 22. The machine-readable medium of claim 21wherein modifying the presentation of the conceptually relatedinformation includes associating particular information with acorresponding user query.
 23. The machine-readable medium of claim 21wherein the search activity information includes all of the informationof a user search engine session.
 24. The machine-readable medium ofclaim 21 wherein the responses received from each of the two or moreusers are selected from the group of search engine activity informationconsisting of query-to-pick information, pick-to-query information,pick-to-pick information, query-to-query information, and combinationsthereof.